Low-power image change detector

ABSTRACT

A sensing device projects near-field spatial modulations onto a closely spaced photodetector array. Due to physical properties of the grating, the point-spread response distributes spatial modulations over a relatively large area on the array. The spatial modulations are captured by the array, and photographs and other image information can be extracted from the resultant data. An image-change detector incorporating such a sensing device uses very little power because only a small number of active pixels are required to cover a visual field.

BACKGROUND

Traditional cameras use a lens or lenses to image each point in a sceneonto a single point on a sensor. In digital cameras, the sensor is atwo-dimensional array of picture elements, or “pixels,” that encodes theimaged scene into digital image data for storage, processing, andreproduction.

Digital imaging has enabled new imaging architectures. Cathey and Dowskitook an early and conceptually important step away from the traditionalmodel by exploiting digital processing. They designed a cubic-phaseoptical plate which, when inserted into the optical path of atraditional camera, led to an image whose (significant) blur wasindependent of the object depth: the image on the sensor plane did not“look good” as it would in a traditional camera. However, subsequentimage processing sharpened the entire blurred image, thus leading toenhanced depth of field. Since then the field of computational imaginghas explored imaging architectures in which the raw signals do notsuperficially resemble a traditional image; instead, the final image iscomputed from such signals. More and more of the total imaging “burden”is borne by computation, thereby expanding the class of usable opticalcomponents. In this way, many optical aberrations can be correctedcomputationally rather than optically. This imaging paradigm has led tonew conceptual foundations of joint design of optics and imageprocessing, as well as a wide range of non-standard imagingarchitectures such as plenoptic, coded-aperture and multi-aperturesystems, each with associated methods of signal processing.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is illustrated by way of example, and not byway of limitation, in the figures of the accompanying drawings and inwhich like reference numerals refer to similar elements and in which:

FIG. 1A is a cut-away view of a sensing device 100 with an odd-symmetrygrating 105 overlying a photodetector array 110, such as a CCD(charge-coupled device) or CMOS (complementarymetal-oxide-semiconductor) sensor.

FIG. 1B depicts sensor 100 of FIG. 1A simulating light incident plane120 at an acute angle 160 to illustrate the sensitivity of curtains 140and foci 145 to the angle of incidence.

FIG. 2 depicts a binary odd-symmetry grating 200 in accordance with oneembodiment.

FIG. 3 depicts a sensing device 300 in accordance with an embodiment inwhich a binary, odd-symmetry phase grating 310 is formed by an interfacebetween materials of two different refractive indices.

FIG. 4A is a plan view of a sensor 400 in accordance with anotherembodiment.

FIG. 4B is a three-dimensional perspective of sensor 400 of FIG. 4A, andshows how light 420 from a direction normal to the grating surface castsan interference pattern 425 on an underlying photodiode array 430.

FIGS. 5A, 5B, 5C, and 5D each depict three boundaries of odd symmetry500 over a two-dimensional photodiode array 505.

FIG. 6 depicts three odd-symmetry gratings 600, 620, and 630, each withfeature segments of different relative widths.

FIG. 7A is a cross-section of a phase grating 700 in accordance with anembodiment that uses more than two levels to produce an odd symmetry.

FIG. 7B is a cross-section of a phase grating 710 that is opticallysimilar to phase grating 700 of FIG. 7A, but uses fewer layers.

FIG. 8 is a cross-section of a phase grating 800 that illustrates howodd symmetry can be extended to curved functions.

FIG. 9 is a plan view of a grating 900 in accordance with an embodimentin which boundaries of odd symmetry 905 extend radially from the centerof the grating, and in which the widths of the feature segments widengradually away from the center.

FIG. 10 is a plan view of a grating 1000 in accordance with anembodiment with concentric boundaries of odd symmetry 1005, and includesa cut-away view along line A-A.

FIG. 11 is a plan view of a grating 1100 in accordance with anembodiment similar to grating 900 of FIG. 9.

FIG. 12 is a plan view of a grating 1200 in accordance with anotherembodiment.

FIG. 13 depicts a grating 1300 in accordance with another embodiment.

FIG. 14 depicts a grating 1400 and associated photodiode array 1405.

FIG. 15 depicts a grating 1500 and associated photodiode array 1505.

FIG. 16 is a plan view of a grating 1600 in accordance with anembodiment with pentagonal boundaries of odd symmetry 1605.

FIG. 17A is a plan view of a grating 1700 in accordance with anotherembodiment.

FIG. 17B depicts the shapes of boundaries 1705 of FIG. 17A.

FIG. 18 depicts a two-dimensional array 1800 of gratings 1805 disposedover a photodiode array (not shown).

FIG. 19 is a flowchart 1900 detailing how an image 1905 is captured andresolved in accordance with grating 1700 of FIG. 17.

FIG. 20 depicts lithographic process for forming an image sensor 2000 inaccordance with one embodiment.

FIG. 21A depicts a camera 2100 in accordance with an embodiment thatincludes a a lens 2105.

FIG. 21B is an example of camera 2100 with a point source 2125 imaged infocus on array 2115.

FIG. 21C is an example of camera 2100 with a point source 2140 imagedout of focus on array 2115.

FIG. 21D is an example of camera 2100 with a point source 2155 imagedmore out of focus than point source 2140 in the example of FIG. 21C.

FIG. 22 is a plan view of a portion of an array of pixels 2200illuminated with three PSFs 2205, 2210, and 2215.

FIG. 23 depicts three spiral PSFs 2300, 2305, and 2310 to illustrate howcameras in accordance with some embodiments can compensate for lensaberrations, including spherical aberration, coma, and Petzval fieldcurvature.

FIG. 24 depicts a tessellated optical element 2400 to illustrate aspectsof phase gratings in accordance with some embodiments.

FIG. 25 depicts how the rich pattern of spatial modulations withinorientation chirps produced by grating 2400 of FIG. 24 facilitatesimproved resolution for out-of-focus PSFs.

FIGS. 26A and 26B depict tessellated gratings 2600 and 2605 inaccordance with some embodiments.

FIGS. 27A and 27B depict tessellated gratings 2700 and 2705 inaccordance with some embodiments.

FIG. 28 depicts a tessellated grating 2800 in accordance with oneembodiment.

FIG. 29 depicts a tessellated grating 2900 in accordance with anotherembodiment.

FIG. 30 depicts a filter array 3000 that can be used in accordance withsome embodiments to produce color images using cameras of the typedetailed in FIGS. 21A-D.

FIG. 31 depicts a color channel 3100, one of four color channels for theembodiment introduced in connection with FIG. 30.

FIG. 32 depicts an image-change detector 3200 that supports a low-powermode.

FIG. 33 depicts array 3205 of FIG. 33 as an array of pixels 3300.

FIG. 34 depicts array 3205 of FIG. 33 with twenty-five nonadjacentpixels 3300 darkened to indicate a subset n2 that is active in anexample of a tentative mode in which an increased number of pixels arepolled for increased motion sensitivity.

FIG. 35 depicts response 1930 of FIG. 19 as an exemplary chirp thatdefines a convex hull 3500.

FIG. 36 is a flowchart 3600 illustrating a method of detecting apparentmotion in accordance with one embodiment of detector 3200 of FIG. 32.

DETAILED DESCRIPTION

The economic pressures for miniaturization of electronic devices,including cameras, arising in the mobile computing market have led tosmaller imager form factors. One technique for miniaturizing imagingarchitectures is one based on integrating diffractive optics withphotodetector arrays. Such architectures can forgo lenses and relyinstead on diffraction gratings that can be created using processessimilar to those used to create the underlying sensor. For a given imageresolution, such diffractive elements enable the construction of imagingdevices much smaller than possible using the optical paths oftraditional cameras, and at a much lower cost.

FIG. 1A is a cut-away view of a sensing device 100 with integrateddiffractive optics. A grating 105 overlies a photodetector array 110,such as a CCD (charge-coupled device) or CMOS (complementarymetal-oxide-semiconductor) sensor. The features of grating 105 offerconsiderable insensitivity to the wavelength of incident light in awavelength band of interest, and also to the manufactured distancebetween grating 105 and photodetector array 110. Grating 105 produces aninterference pattern for capture by array 110. Photographs and otherimage information can then be extracted from the pattern.

Light in a wavelength band of interest—such as the visible spectrum—isincident grating 105 from a direction 115 that is normal to a transverseplane 120 of the grating 105. Dashed lines 125 highlight periodicboundaries of substantially odd symmetry. Each of these boundaries is aresult of features 130 and 135 of odd symmetry, and produces a normallyarranged curtain 140 of minimum intensity created by destructive phaseinterference between adjacent features 130 and 135. Curtains 140 areseparated by foci 145, and the collection of curtains 140 and foci 145(curtains of maximum light intensity) extend from grating 105 throughthe body 150 of device 100 to produce an interference pattern onphotodetector array 110. In this illustration, the pattern of intensityvariations evident in the foci and curtains are near-field spatialmodulations that result from near-field diffraction. One photosensitiveelement 155 within array 110 is shaded beneath a focus 145 to serve as areference for a subsequent discussion of the sensitivity of device 100to the angle of incident light.

The image of FIG. 1A resulted from a simulation of a sensing device withthe following parameters and assuming specific parameters. Body 150 isof fused silica, and is in contact with a conventional photodetectorarray 110 with photosensitive elements spaced by 2.2 μm. The top ofgrating 105 is an air interface in this example. The relatively smallsegments of features 130 and 135 are about 1 μm, and the relativelylarger segments are about 4 μm. These segments generally form transverseplane 120, which is separate from array 110 by about 25 μm. Curtains 140and foci 145 are the destructive and constructive interference patternsfor 532 nm incident light.

The thickness of body 150 and lengths of the segments of features 130and 135 were optimized for 400 nm light despite the selection of 532 nmlight for the simulation. As a consequence, the tightest focus occursabout 5 μm above array 110 (at the 20 μm mark). The resultant curtains140 plainly separate foci 145 well above and below the 20 μm mark,however, illustrating a robust insensitivity to wavelength within theband of interest. The relatively deep and continuous penetration ofcurtains 140 also provides considerable manufacturing tolerance for thethickness of body 150. These advantages obtain because the near-fieldspatial modulations projected onto array 110 are wavelength independentover the wavelength band of interest, which means that the adjacentmodulations (dark and light) do not reverse signs with changes inwavelength within the band of interest.

FIG. 1B depicts sensor 100 of FIG. 1A simulating light incident plane120 at an acute angle 160 to illustrate the sensitivity of curtains 140and foci 145 to the angle of incidence. Using element 155 as a referencepoint, we see that that the foci 145 that illuminated element 155 inFIG. 1A has considerably moved to the right in FIG. 1B. Curtains 140 andfoci 145 extend at an acute angle that relates to angle 160 according toSnell's law. The separation of foci 145 by curtains 140 is maintained.Sensor 100 is thus sensitive to the angle of incidence.

FIG. 2 depicts a binary odd-symmetry grating 200 in accordance with oneembodiment. Each of three boundaries of odd symmetry is indicated usinga vertical, dashed line. The upper features of grating 200 are at aheight sufficient to induce one half wavelength of retardation in theband of interest relative to lower features, or π radians of relativephase delay. Features 205 and 210 on either side of each boundaryexhibit odd symmetry with three differently sized segments W₀, W₁, andW₂. With this arrangement, paired segments (e.g., W₀ within features 205and 210) induce respective phase delays that differ by approximatelyhalf a wavelength over the wavelength band of interest.

FIG. 3 depicts a sensing device 300 in accordance with an embodiment inwhich a binary, odd-symmetry phase grating 310 is formed by an interfacebetween materials of two different refractive indices, a polycarbonatelayer 315 and optical lanthanum dense flint glass 320 in this example.Each of four boundaries of odd symmetry 325 is indicated using avertical, dashed line. As in the foregoing examples, the upper featuresof grating 310 induce phase retardations of half of one wavelength (πradians) relative to lower features. Features 330 and 335 on either sideof each boundary exhibit odd symmetry. With this arrangement, pairedfeatures induce respective phase delays that differ by approximatelyhalf a wavelength over the wavelength band of interest.

These elements produce an interference pattern on an analyzer layer 327(e.g., a conventional photodiode array) in the manner detailed inconnection with FIGS. 1A and 1B. This example assumes light incident thelight interface of grating 300 is normal to the transverse plane ofphase grating 310, in which case light fields that enter grating 310equidistant from a one of the boundaries of odd symmetry 325, such as atlocations (−X,0) and (X,0), are out of phase at points beneath array 310(e.g., point (0,Z)), and thus destructively interfere to producecurtains of minimum intensity (e.g., curtains 140 of FIG. 1). Neitherthe depth Z nor the wavelength of light over a substantial spectrumsignificantly influences this destructive interference. Constructiveinterference similarly produces foci of maximum intensity (e.g., foci145 of FIG. 1). Both the high and low features admit light, whichprovides relatively high quantum efficiency relative to gratings thatselectively block light.

The following discussion details phase gratings in accordance withexamples described by Patrick R. Gill and David G. Stork in an upcomingpaper. “Lensless Ultra-Miniature Images Using Odd-Symmetry Spiral PhaseGratings.” © 2013 Optical Society of America. In that article, Gill andStork describe a phase grating formed by a high-n, low-dispersionsubstrate and a low-n, high-dispersion coating that can introduceapproximately independent phase shifts in all normally incident visiblelight. Similar gratings are discussed above. If there exist certainpoints p on this interface that satisfy the following symmetry in theirtransmission t(·) and phase retardation φ(·),

t(p+y)=t(p−y)∀y  (1)

φ(p+y)=φ(p−y)+π°2nπ∀y,nεI  (2)

where y is a horizontal translation transverse to the grating direction,then the grating has odd symmetry about points p, and light willinterfere destructively below p, regardless of λ and depth z.

A linear odd-symmetry grating above a photosensor array could passinformation from a single spatial orientation of features in the farfield (transverse to the grating orientation). However, to captureinformation about arbitrarily oriented features of a complex scene, itis preferable to have a complete distribution of orientations in thediffractive optic. More generally, if the point-source responses (PSRs)are approximately spatially invariant, the transfer function of theimager approximates convolution with the PSR function. In such a case,the PSR should have significant power at all 2D spatial frequencies tomake the inversion problem of image recovery well-conditioned.

In one example provided in Gill and Stork, gratings were numericallyoptimized to focus visible light onto a photodetector array 100 μmbelow. Optical simulations estimated the imaging performance of such adevice from a 60×60 pixel array with 2.2 μm pitch 100 μm below thegratings with the sensor illuminated by a complex scene far (>>100 μm)from the sensor. The resultant photocurrent from the pixel array wasunintelligible; however, the scene was reconstructed to a higherresolution than possible using a much larger diffractive imagers basedon Talbot-effect angle-sensitive using Tikhonov regularization. Gill andStork report that compressed sensing techniques could be applied toimprove the reconstruction quality if the scene is known to have acompressible structure. Compressed sensing could be especiallyadvantageous if small gaps in the Fourier transform of the PSR exist.

FIG. 4A is a plan view of a sensor 400 in accordance with anotherembodiment. Relatively high segments 405 and low segments 410 on eitherside of each of eight boundaries of odd symmetry 415 create a grating inwhich the widths of the segments increase with distance from the centerof the sensor. For a given focal depth, light of higher frequenciestends to produce a sharper focus with narrower feature widths. Sensor400 can therefore be optimized such that the central portion of thegrating is optimized for collection of relatively higher frequencylight, and the peripheral area for collection of relatively lowerfrequency light. This topic is detailed below in connection with otherFigures.

FIG. 4B is a three-dimensional perspective of sensor 400 of FIG. 4A, andshows how light 420 from a direction normal to the grating surface castsan interference pattern 425 on an underlying photodiode array 430.Curtains and foci, as detailed previously, respectively cast shadows 435and bright shapes 440 to be sensed by individual photosensitive elements445 of array 430. Array 430 captures a digital representation of pattern425.

FIGS. 5A, 5B, 5C, and 5D each depict three boundaries of odd symmetry500 over a two-dimensional photodiode array 505. Curtains 510 castshadows 515 on the underlying photodetectors 520, and the patterns thuscreated are different depending upon the angle of incident light. Array505 can therefore sample the resultant interference pattern to obtaininformation as to the angle of incidence.

FIG. 6 depicts three odd-symmetry gratings 600, 620, and 630, each withfeature segments of different relative widths. It can be useful tocreate a sensor with multiple width ratios, as shown, to compensate formanufacturing tolerances that impact the relative heights of the gratingfeatures. Assuming, for example, that grating 600 is width optimized fora manufacturing process of interest, but that the process produces arelative phase delay of 40% rather than the ideal 50% to form curtainsof minimum intensity at the desired positions. To a first order theincreased width of the relatively wide segments, as depicted in grating630, can improve the distortion resulting from the erroneous phaseoffset. Phase offsets above 50% can be corrected for by narrowing therelatively wide segments, as depicted in grating 620. Some embodimentsinclude a mixture of relative segment widths covering different areas ofa photodiode array to accommodate manufacturing tolerances. Imagesassociated with the gratings that provide the sharpest focus, or thesharpest focus for a wavelength of range of wavelengths, can be selectedor combined to obtain the desired image data. The different gratings mayalso perform better for light of different wavelengths or incidentangles, so selection of which gratings to use for a given image may beoptimized for variables other than manufacturing tolerances.

FIG. 7A is a cross-section of a phase grating 700 in accordance with anembodiment that uses more than two levels to produce an odd symmetry.Additional levels may allow for sharper focus, but may require morecomplex manufacturing processes. If gratings are to be made usingphotolithography, for example, additional levels require additional masksteps. Paired surfaces on either side of each boundary of odd symmetryintroduce respective paired phase delays that differ by approximatelyhalf a wavelength, plus an integer number of wavelengths, over thewavelength band of interest.

FIG. 7B is a cross-section of a phase grating 710 that is opticallysimilar to phase grating 700 of FIG. 7A, but uses fewer layers. Theresultant larger abrupt discontinuities 715 may introduce undesirableimage artifacts or may be difficult to manufacture accurately, but thereduced number of levels may reduce manufacturing costs.

FIG. 8 is a cross-section of a phase grating 800 that illustrates howodd symmetry can be extended to curved functions.

FIG. 9 is a plan view of a grating 900 in accordance with an embodimentin which boundaries of odd symmetry 905 extend radially from the centerof the grating, and in which the widths of the feature segments widengradually away from the center. Grating 900 captures image informationat sixteen discreet angles with a continuously variable set of widths.While convenient to draw grating 900 as a circle, other shapes may beused. In some embodiments, for example, collections of gratings arearrayed over a photodiode array. In such cases grids that share commonboundaries (e.g., such as hexagonal, square, or triangular boundaries)make more efficient use of the underlying photodiodes.

FIG. 10 is a plan view of a grating 1000 in accordance with anembodiment with concentric boundaries of substantially odd symmetry1005, and includes a cut-away view along line A-A. In this example thewidths of the feature segments are discrete and the angles arecontinuous. The spacings of grating 1000 appear consistent, but may bevaried to allow for sharp focus for a range of wavelengths, angles ofincidence, or manufacturing variations.

FIG. 11 is a plan view of a grating 1100 in accordance with anembodiment similar to grating 900 of FIG. 9. The two halves of grating900 provide essentially the same information. Grating 1100 addshalf-circle polarization filters 1105 and 1110 with perpendicularorientations. Each half of grating 1100 thus produces image dataspecific to one of two polarizations, and these data can be usedseparately or together. More or fewer filters, with the same ordifferent orientations, may be used in other embodiments. Differenttypes of filters can also be used to cover all or a portion of gratingsof the type described herein.

FIG. 12 is a plan view of a grating 1200 in accordance with anotherembodiment. Curved boundaries of odd symmetry 1205 extend radially fromthe center of the grating, and the widths of the feature segments widengradually away from the center. The curvature of boundaries 1205 providecontinuously varying angular information similar to what is availablefrom grating 1000 of FIG. 10 while retaining the continuously varyingspacings of grating 900 of FIG. 9.

FIG. 13 depicts a grating 1300 in accordance with another embodiment. Asnoted previously, different widths of the grating features providesharper focus for different colors of light within the wavelength bandof interest. Grating 1300 has the same radial symmetry of grating 900 ofFIG. 9, but those areas for which the spacing is optimized for blue,green, and red light are provided with filters to admit their respectivewavelengths. Omitting wavelengths that provide a blurred interferencepattern on the underlying analyzer can improve image sharpness, and canallow more accurate reconstruction of color image data. Grating 1300 isbounded by an opaque mask 1305 that defines the limit of the aperture.

FIG. 14 depicts a grating 1400 and associated photodiode array 1405.Grating 1400 has parallel odd-symmetry boundaries 1410, which may havefeatures of the same or different widths, or of varying widths along oneor more boundaries. Parallel boundaries with the requisite diversity ofwidths and spacings to sample a sufficient number of spatial frequenciescan image one-dimensional images, e.g., barcodes. Array 1405 is shownalongside, rather than below, grating 1400 to highlight the angle θ_(A)between the direction of boundaries 1410 and the columns ofphotosensitive elements in array 1405. Angle θ_(A) creates morediversity of measurements because the linear shadow covers differentpercentages of pixels in different rows. In one embodiment angle θ_(A)is selected so that the top of each boundary is offset from the bottomby about one pixel of array 1405.

FIG. 15 depicts a grating 1500 and associated photodiode array 1505.Grating 1500 has parallel, right-angled boundaries 1510, which may havefeatures of the same or different widths, or of varying widths along oneor more boundaries. Parallel boundaries with the requisite diversity ofwidths and spacings along two dimensions to sample a sufficient numberof spatial frequencies can image e.g. point sources, such as to identifythe position of the sun, a fiducial LED, or a retroreflective element orpatch used for motion capture. Angle θ_(A) can be introduced for thereasons presented above in connection with FIG. 14. Point sourceidentification may also be accomplished with a grating that is alsosuitable for an imaging function.

FIG. 16 is a plan view of a grating 1600 in accordance with anembodiment with pentagonal boundaries of odd symmetry 1605. In thisexample the widths of the feature segments are discrete, but they canvary along one or more boundaries in other embodiments. Straightboundaries may be advantageous over curved ones because line segmentscan more easily provide precise odd symmetry.

Grating 1600 provides information at five different orientations. Otherboundary shapes, such as other polygons, are used in other embodiments.In general, polygons with odd numbers of sides provide greaterorientation diversity than polygons with a similar but even number ofsides (e.g., a pentagon provides more orientation diversity than asquare or a hexagon).

FIG. 17A is a plan view of a grating 1700 in accordance with anotherembodiment. Recalling that relatively narrow (wide) segment spacingworks better for relatively high (low) frequencies, feature spacingincreases along odd-symmetry boundaries (between dark and light regions)with distance from the center. Curved boundaries of odd symmetry 1705extend radially from the center of the grating to the periphery,radiating out between the dark (elevated) and light (recessed) arms nearthe center. The curved boundaries are obscured by grating features inFIG. 17A, so the shapes of boundaries 1705 are depicted in FIG. 17B forease of review.

The segment widths do not continue to increase with radius, as there isa maximum desired width for a given wavelength band of interest (e.g.,the widest may correspond to the lowest frequency of visible red light).The features that define boundaries 1705 therefore exhibitdiscontinuities as they extend toward the periphery of grating 1700. Inthis example, grating 1700 has three discrete areas each tuned to asubset or all of the wavelengths in the band of interest.

FIG. 18 depicts a two-dimensional array 1800 of gratings 1805 disposedover a photodiode array (not shown). Each of gratings 1805 is identical,but any number of parameters, many of which are discussed previously,can be varied within and among gratings 1805. For example, differentshapes and types of gratings can be used to create and image differenttypes of interference patterns that can be combined or used separatelyto obtain some desired result. The decision to consider all or aspecific subset of information generated by one or more of theconstituent gratings can be done once, such as at time of manufacture toaccommodate process variations, or can be done dynamically to highlightdifferent aspects of a scene. Emphasizing aspects of different patternscan be used, for example, to highlight light of different polarizations,wavelengths, or angles of incidence.

Spaced gratings facing the same direction, particularly when theircharacteristics are well matched, can be used to sense moving objects.Assuming matched gratings with a fixed separation receiving light fromthe same scene, the difference between the photocurrents of therespective analyzer layers is sensitive only to objects relatively closeto the pair. Further, the time derivative of this difference issensitive to nearby, moving objects, and is insensitive to relativelydistant moving or stationary objects.

FIG. 19 is a flowchart 1900 detailing how an image is captured andresolved in accordance with grating 1700 of FIG. 17. First, an image1910 is presented such that light from image 1910 is incident grating1700. The incident light passes through phase grating 1700 to produce anintensity pattern 1920 on an underlying two-dimensional array ofphotosensors (not shown), which captures the pattern (1915). Thecaptured pattern 1920 may appear unintelligible to a human; however,because grating 1700 has sharp features in its point-spread function(PSF), the pattern contains rich information about the image.

The PSF of grating 1700, possibly in combination with the underlyingarray, is known from a prior calibration or high-fidelity simulation.The way in which the PSF varies as a function of incident angle andcolor may also be similarly determined. This information is representedby a response 1930. A mathematical conversion based on this response canthus be used to reconstruct image 1910 from pattern 1920.

To recover the original image, responses 1920 and 1930 are combined toform an inverse problem (1925), which is solved (1935) to recover aversion 1940 of the original image. One embodiment employs thewell-known Tikhonov regularized inversion technique to accomplish steps1925 and 1935. Take as a starting point a) detailed knowledge of the PSFof grating 1700, b) knowledge of the noise level of the system undercurrent illumination conditions, and c) the specific readings observedfor this image (pattern 1920). Express the unknown image as an N×1vector x, where N is the total number of pixels one wishes toreconstruct. Express the readings from the photosensor as an M×1 vectory, where M is the total number of photosensors in the array. Expressdetailed knowledge of the PSF as an M×N matrix A such that for any imagex, the formula yielding expected observed signal y under x is y=Ax,called the “forward equation.”

To reconstruct an image, it suffices to solve the forward equation witha known measurement vector y for an unknown image x as follows. Multiplyboth sides of the forward equation by A^(T) (the transpose of A) toobtain A^(T) y=A^(T) Ax. The matrix A^(T) A is square and in principlecould be directly inverted to recover x; however usually this inversionis poorly conditioned when noise is present and when not alleigenvectors of A^(T) A have equally large associated eigenvalues. Thusin practice, Tikhonov regularization (as follows) usually deliverspreferable results.

Next, select a regularization parameter λ>0 based on the noise level atthe current illumination conditions. Finally, invert the matrix (A^(T)A+λI) (where I is the identity matrix), assume (A^(T) A+λI)≈(A^(T) A)and multiply on the left of the preceding equation to obtain:

x≈(A ^(T) A+λI)⁻¹ A ^(T) y  (3)

Therefore, for a given regularization parameter λ, the image recoveredthrough Tikhonov regularization is a linear combination of the readingsfrom the photosensor. If the PSF is sufficiently spatially invariant tothe extent that its spatial dependence can be neglected, thesecomputations can be done in the Fourier domain, allowing for much fasternumerics.

Another embodiment recovers the matrix x using compressed sensing. Ifthe scene is expected to be sparse in some basis (such as a wavelettransform W for natural images), the following methodology can be used.We can recover the sparse scene components z where x=Wz by finding the zthat minimizes the following cost function: ½ r^(T)r+λf(z), where r isthe residual (y−AWz), λ>0 is a regularization parameter (different fromthat used in Tikhonov regularization, but also noise-dependent), andf(z) is a function penalizing non-sparse z. If f(z) is a convex functionof z such as the L₁ norm, this optimization problem can be solvedefficiently using convex optimization techniques. The penalty functionf(z) can also take on other forms, including terms penalizing totalvariation in the reconstructed image x or other prior scene knowledge.

Some of the chief advantages of compressed sensing over linearapproaches such as Tikhonov regularization are that the former allowmore prior information about the expected scene structure to help shapethe final image. Further, if A^(T) A does not have full rank or cannotmeasure certain aspects of the scene (for example, due to some near-zeroregions of the 2D Fourier transform of the PSF), using compressedsensing sometimes overcomes these limitations given correct priorinformation about the expected images.

The foregoing Tikhonov and compressed-sensing techniques can includeiterative methods to reduce problem complexity. For example,Richardson-Lucy deconvolution can iteratively approximate Tikhonovregularized inversion and iterated wavelet thresholding can be anumerically efficient way to converge to a compressed-sensing-likesolution.

In some embodiments the purpose of the sensor is not to reconstruct animage, but to perform some optical sensing task. In such cases thevector x may represent the sought measurement rather than the field ofimage pixels, and the forward transform A can be appropriately modified.

FIG. 20 depicts a lithographic process for forming an image sensor 2000in accordance with one embodiment. First, a wafer 2005 of material thatis transparent over the wavelength band of interest is patterned with amask 2010 that defines the relatively high features of what will becomean odd-symmetry grating surface of the type detailed herein. Next, theexposed surface of wafer 2005 is etched to create recessed regions 2015.Mask 2010 is then removed. Finally, wafer 2005, now comprising agrating, is bonded to a photodiode array 2025. Photolithographic andwafer-bonding processes are well known to those of skill in the art, soa detailed discussion is omitted.

FIG. 21A depicts a camera 2100 in accordance with an embodiment in whicha converging optical element, in this case a lens 2105 (although asingle-element lens is shown for simplicity of illustration, generallythe optical element can comprise one or more refractive, diffractive,and/or reflective elements), is used in conjunction with a phase gratingelement, grating 2110, disposed in the path between the optical elementand a dense photodetector array 2115 to form images thereon. A sceneincident the front side of lens 2105 is projected through grating 2110and onto array 2115. Grating 2110 induces spatial modulations in theincoming light and passes the resulting interference pattern to array2115, which captures a digital representation of the spatialmodulations. An integrated processor 2120 electrically coupled to array2115 computes an image of the scene from the digital representation. Theprocessor is shown also physically coupled to array 2115, but theprocessor can be located elsewhere in other embodiments.

Lens 2105 defines a front focal point FFP and a rear focal point RFP,and is spaced from grating 2110 by a distance less than the image-planedistance D between lens 2105 and focal point RFP. Array 2115 is on theopposite side of focal point RFP from grating 2110 in this example.Grating 2110 may be an odd-symmetry grating that has properties detailedabove in connection with the preceding figures. In other embodiments(such as an embodiment primarily operating in a macro mode) the focallength of lens 2105, defined for objects at infinity, may be closer tolens 2105 than to grating 2110, or may move over a range thatencompasses such relative positioning.

Surface features of grating 2110 are separated from array 2115 by adistance X. Though shown as separate structures for ease ofillustration, grating 2110 can be integrated with or attached to array2115. Distance X in camera 2100 is, in this example, no more than 400times a longest wavelength of interest λ_(max) in the medium(s) betweenthe surface features of grating 2110 and array 2115 (X≦400λ_(max)). Forexample, a camera in which λ_(max) is 0.5 microns may have a spacing Xbetween the features of grating 2110 and the surface of array 2115 of upto 200 microns.

FIG. 21B is an example of camera 2100 with a point source 2125,represented by tip of an arrow, that is imaged in focus on array 2115.Grating 2110 is out of the focal plane, so the light from lens 2105presents a blur-spot PSF 2130 to grating 2110. (As in other examplesused herein, the area occupied by PSF 2130 refers to the area of thecentral lobe.) Grating 2110 produces an interference pattern fromfunction 2130, but the illumination boundaries of the pattern are notevident in the tightly focused, diffraction-limited spot 2135 on array2115. Objects at the range and position of point source 2125 are tightlyfocused (field curvature and other aberrations may change the best focusrange for other positions), and are nominally imaged at the fullresolution of array 2115, assuming lens 2105 is capable of suchresolution.

FIG. 21C is an example of camera 2100 with a point source 2140 that isimaged out of focus on array 2115. As in the prior example, the lightfrom lens 2105 presents a blur-spot PSF 2145 to grating 2110, andgrating 2110 produces a pattern of spatial modulations. Because pointsource 2140 is imaged out of focus, however, the area of PSF 2150 atarray 2115 is greater than in the example of FIG. 21B, and illuminationtransitions/substructure within the pattern area are evident. In camera2100, these illumination transitions are near-field spatial modulationsinduced by features of grating 2110. The resultant spiral pattern of PSF2150 is preferably an invertible orientation chirp. As used herein, an“orientation chirp” is a pattern of spatial modulations that coverranges of spatial frequencies and orientations sufficient to recover animage at a desired resolution.

FIG. 21D is an example of camera 2100 with a point source 2155 that isimaged more out of focus than point source 2140 in the example of FIG.21C. Light from lens 2105 presents a blur-spot PSF 2160 that is stillgreater than PSF 2145, and a resultant invertible PSF 2165 on array 2115is similarly larger than PSF 2150. Although not shown, imaging a pointsource at the FFP of FIG. 21A produces an invertible PSF includingorientation chirp features. Two point sources, one in front of and onebehind point 2125 but along the same optical axis, may producesimilar-sized orientation chirps. Due to aberrations in the lens system,however, the chirps may differ—such differing characteristics may beused to resolve range, as detailed further below.

FIGS. 21A-D illustrate the general point that the pattern area and therichness of the accompanying spatial modulations on array 2115 are afunction of focus, the duller the focus the greater the area and thebetter resolved the spatial modulations. Point sources farther away fromlens 2105 than point source 2125 of FIG. 21A produce ever larger PSFs onthe array as they move away from (or towards) lens 2105.

The PSF for an out-of-focus point source is a scaled version of anorientation chirp from grating 2110, where the diameter of theorientation chirp is proportional to defocus of the point source. Theobservations at the sensor plane (the surface of array 2115) willtherefore be the in-focus and out-of-focus imaged points, each convolvedwith the orientation chirp at a chirp phase dependent upon the positionthe light ray bundle received from that point, scaled according to anout-of-focus parameter, and spatially superimposed with likecontributions from other imaged points. Camera 2100 can recoverrelatively high-resolution images of out-of-focus objects because thisconvolution is computationally invertible for the majority of commonimage capture situations. In this context, “computationally invertible”means that image data can be recovered to a specified degree ofprecision using e.g. inverse, pseudoinverse, and compressed-sensingtransformations. A PSF is computationally invertible, for example, ifits 2D Fourier transform is “complete,” or has substantial amplitude atall spatial frequencies required to recover an image at a specifiedresolution.

Not all spiral PSFs are complete. For example, Archimedean spirals haveregularly spaced arms whose Fourier transforms have peaks at thereciprocal of the inter-arm period and nulls between these peaks. Incontrast, the spiral PSR 1930 of FIG. 19 has few, unevenly spaced armsthat are sharply bounded and sweep through all orientations, so it hassignificant Fourier power at all spatial frequencies and is complete.Due to this completeness, accurate deconvolution is relativelywell-conditioned, so undoing the effect of the PSF is relativelystraightforward. Regardless of whether computations are performed in theFourier domain or the spatial domain, deconvolution works well if theFourier transform of the PSF has no zeros. In the case that a pointsource causes a blur spot 2160 that is not concentric with a spiral, theresulting PSF will contain a spatially wrapped version of the PSF.Spatially wrapping the spiral does not substantially affect itscompleteness.

Camera 2100 can measure light intensity from photodetector array 2115without first needing to focus (although some embodiments can focusmanually or automatically). Data captured by array 2115 includesorientation chirps with Fourier-component strengths that vary with depth(see FIGS. 21B-D). The Fourier transform of the local observations willbe the product of the imaged object's Fourier transform and thedepth-dependent Fourier transform of the orientation chirp. By searchingfor the depth-specific kernel that best matches this product for eachimaged point, scene depth can be determined, assuming the scene has sometexture, as detailed below.

The depth d of a local scene patch x can be inferred from readings ythrough Bayesian estimation as follows. First, a likelihood p(y|d) ofeach depth can be computed by a further Bayesian estimation based onknowledge that the Fourier transform of y is the product of the Fouriertransforms of x and the depth-dependent PSF, and with knowledge oftypical power spectra of photographed objects. Next, this likelihoodp(y|d) is weighted by a Bayesian prior on the known distribution ofdepths and depth changes in a scene to arrive at a posterior probabilityof p(d|x) for depth at each point in the scene. Bayesian estimation ofthe depth map of a scene based on depth and depth change priors, as wellas point-wise estimates of depth associated with corresponding certainty(indicated by the height of peaks in the likelihood p(y|d)) is atechnique known to those skilled in the art, and will not be furtherdiscussed here. In this application, knowledge of the true depth map isimportant for accurate image recovery (to be described shortly)precisely for those images that have significant Fourier power inspatial frequencies that interact with the Fourier transform of the PSF.Thus, accurate depth maps are possible where the scene has fine texture,and where scene patches lack this texture convolution with the PSF doesnot degrade image quality in the scene.

Next, the Fourier transforms are deconvolved in image space or theFourier domain; the problem scale will dictate which of these is faster.The deconvolution kernel can also be made to vary with light level for aWeiner-optimal reconstruction (although humans tend to preferoverly-sharpened images; this sharpening filter can be incorporated withthe deconvolution filter to save an additional step).

The result of selecting the correct filter followed by deconvolution isa depth map and a reconstruction of the original image. If theorientation chirp is Fourier-complete, the reconstructed image canresolve the same number of pixels as array 2115. This is unlike mostplenoptic cameras, and is made possible by the fact that each pixelreading contributes useful information to the deconvolution problem. Inthe case where a PSF's high-frequency components are small, processor2120 may smooth the highest spatial frequencies to avoid adding too muchnoise. In low-light conditions, camera 2100 may lose e.g. a factor oftwo in resolved pixels due to this effect; this represents animprovement over existing plenoptic cameras, whose pixel efficiency maybe as low as 4%. For well-formed orientation chirps according to anembodiment and general imaging conditions, PSFs with a central lobediameter up to six photodetector pitches should be invertible to recoverimage features with a spatial frequency up to at least 0.25 cycles perphotodetector (Nyquist frequency being 0.5 cycles per photodetectorpitch in the major dimensions of the photodetector array). Suchperformance depends in part on the lens element having a sufficientmodulation transfer function at the relevant spatial frequencies.

FIG. 22 is a plan view of a portion of an array of pixels 2200illuminated with three PSFs 2205, 2210, and 2215. PSF 2205 is anorientation chirp representing a sharply focused point source;illumination substructure cannot be resolved given the pitch of array2200. If all points of a given scene are in focus, image resolution isprimarily a function of array pitch, or of array pitch and the diameterof a diffraction-limited spot.

PSF 2210 is an orientation chip representing a poorly focused pointsource; spatial modulations appear as spiral arms of a computationallyrich PSF that can be resolved by array 2200 to locate the correspondingpoint source in the image. Finally, PSF 2215 represents a point sourcewhose focus is between those of PSFs 2205 and 2215; spatial modulationscan again be resolved to locate the corresponding point source in theimage.

For both PSF 2210 and 2215, the resolution of the image is limited bythe larger of the pitch and the spacing of the separation between armsof the PSF spiral. In this illustration, the three point sources areeasily located in the two dimensions of array 2200. Further, the threedisparate pattern areas of the three PSFs provide a measure of distancein a dimension normal to array 2200. Cameras like camera 2100 of FIGS.21A-D can therefore provide extended depths of field, focused images forout-of-focus objects, and measures of distance from image data.

FIG. 23 depicts three spiral PSFs 2300, 2305, and 2310 to illustrate howcameras in accordance with some embodiments can compensate for lensaberrations, including spherical aberration, coma, and Petzval fieldcurvature. Such compensation can simplify primary lens design and allowan increase in aperture without sacrificing image quality.

Spherical aberration is the condition whereby the focal length of agiven annulus of a lens varies linearly with the annulus' radius. In theconfiguration of FIG. 21, this condition may influence the shapes oforientation chirps on the array. PSF 2300 of FIG. 23 is a hypotheticalideal chirp, the result of a perfect lens. PSF 2305 shows a type ofchirp distortion that may result from a lens with spherical aberration.As compared with PSF 2300, PSF 2305 has relatively linear arms near thecenter. So long as the orientation chirp is complete (invertible torecover the image data), imaging performance will not be degraded. Evenif not complete, imaging performance may be acceptable if theorientation chirp is sufficiently invertible to recover images to adesired resolution.

A lens has coma if light passing through different annuli centered onthe lens forms annuli on the image sensor whose center varies withannulus radius. As shown in PSF 2310, coma produces an elongated anddistorted, but complete spiral. Petzval field curvature is theaberration whereby the lens' focal surface is not planar. As withspherical aberration, coma, Petzval field curvature, and otheraberrations can be undone if the orientation chip is sufficientlycomplete.

Lens aberrations can be beneficial in some embodiments. A PSFout-of-focus to one side of the image plane can cover a pattern area ofthe same size as a PSF out-of-focus to the other side of the imageplane. If two such PSFs are identical, then the camera may not be ableto distinguish between them. Lens aberrations can render such PSFsdistinguishable, however, such as by producing opposite asymmetries, andcan therefore allow cameras in accordance with some embodiments tobetter distinguish point sources along the axis of incidence.

FIG. 24 depicts a tessellated optical element 2400 to illustrate aspectsof phase gratings in accordance with some embodiments. Element 2400 istessellated with spiral ensembles 2410 of sub-elements 2405—depicted ascurvilinear boundaries—that are contiguous across tessellation borders(the hexagonal borders are for illustration, and do not representphysical structure in this example). The sub-elements of each ensembleare arranged such that light converged by element 2400 from a pointsource and passing through one of ensembles 2410 forms a PSF withspatial modulations representative of the ensemble. In one aspect, thetessellated optical element further converges what would otherwisestrike a sensor array as a blurry PSF into a PSF that, while of similarsize to the hypothetical PSF, contains high-frequency substructure.

Returning for a moment to the example of FIG. 21D, the blur spot PSF2160 is assumed to be centered on an ensemble of spiral features toproduce the spiral PSF 2165. This is a somewhat special case. Pointsources at the same distance from the camera yet in general positionwill have a PSF containing all sub-elements 2405 of at least oneensemble 2410 collected from neighboring ensembles, with some of themspatially wrapped around. In the example of FIG. 24, a PSF outline 2415represents the area of a central lobe that is off center with respect toany of the sub-gratings 2410, but that nevertheless covers enoughgrating features 2405 to produce an invertible orientation chirp. Ingeneral, it is beneficial that the wrapping of spatial features betweenensembles 2410 not substantially alter the magnitude of the componentsof the Fourier transform of the resultant orientation chirp. A circlelike outline 2415, of sufficient area to encompass one of ensembles2410, can be swept along a path between neighboring ensembles while, forall intermediate circle positions along the swept path, the swept circlecontains optical sub-elements arranged at all the orientations containedin the circle at the start of the path (e.g., all positions producesimilar spectra, but with shifting phase).

FIG. 25 depicts how the rich pattern of spatial modulations withinorientation chirps produced by grating 2400 of FIG. 24 facilitatesimproved resolution for out-of-focus PSFs. As in other examples, thepreceding digit or digits of each element name indicate the figure inwhich the element was introduced. Using this convention, elements 24##and 25## refer to features depicted in FIGS. 24 and 25, respectively.

In the top row of FIG. 25, light rays from a point source 2500 passthrough a lens (not shown) and onto tessellated grating 2400 of FIG. 24over the area 2505 encompassed by outline 2415 as blurred PSF 2510. Thegrating creates orientation chirp 2515, which includes a rich set ofspatial modulations as a consequence of the sub-elements 2405 within thearea of outline 2415. Chirp 2515 is not a spiral because PSF 2510 is notcentered on an ensemble (the pattern of chirp 2515 is shown to match thepattern of sub-elements 2405 within the PSF outline 2415). However, thespatial modulations of chirp 2515 are sufficient that chirp 2515 isinvertible.

The second row of FIG. 25 is similar to the first, but includes lightrays from adjacent point sources 2520 that illuminate overlapping areas2525 to produce a pair of overlapping, blurred PSFs 2530. The gratingcreates a discernible pair of orientation chirps 2535 and 2540, thelocations of which can be computationally inverted to a higherresolution than could the smeared PSFs 2530. Chirps 2535 and 2540, shownseparately to the right, are slightly different from one another becauseeach PSF 2530 impinges upon a slightly different area of the grating.

The third row of FIG. 25 shows a constellation of nine point sources2542 that illuminate an area 2545 on the grating with overlapping,blurred PSFs 2550, and the resultant nine orientation chirpscollectively labeled 2555. As in the last example, the locations of thepoint sources corresponding to the orientation chirps 2555 can beresolved with far greater precision than could be accomplished using thePSFs 2550.

FIGS. 26A and 26B depict tessellated gratings 2600 and 2605 inaccordance with some embodiments. Grating 2600 is depicted usingboundaries between high and low features, whereas grating 2605 depictsrelatively high and low features in black and white, respectively.

FIGS. 27A and 27B depict tessellated gratings 2700 and 2705 inaccordance with some embodiments. Grating 2700 is depicted usingboundaries between high and low features, whereas grating 2705 depictsrelatively high and low features in black and white, respectively.

FIG. 28 depicts a tessellated grating 2800 in accordance with oneembodiment. Grating 2800 depicts relatively high and low features inblack and white, respectively.

FIG. 29 depicts a tessellated grating 2900 in accordance with anotherembodiment. Grating 2900 depicts relatively high and low features inblack and white, respectively.

FIG. 30 depicts a filter array 3000 that can be used in accordance withsome embodiments to produce color images using cameras of the typedetailed in FIGS. 21A-D. Filter array 3000 includes four color filters,a red filter 3005, two green filters 3010 and 3015, and a blue filter3020. Each filter is associated with what amounts to an instance of acamera like camera 2100 of FIG. 21A that acts as one of four colorchannels for the overall camera. For each camera like camera 2100, thewavelength band of interest is limited to the wavelengths passed by thecolor filter in the optical path.

FIG. 31 depicts a color channel 3100, one of four color channels for theembodiment introduced in connection with FIG. 30. Channel 3100 issimilar to camera 2100 of FIGS. 21A-D, so a detailed discussion isomitted. Briefly, channel 3100 includes a color filter, a lens 3105whose optical properties should be tuned for the light frequenciesadmitted by the color filter, a grating 3110, and a photodetector array3115. The red filter 3005 of FIG. 30 is inserted somewhere in theoptical path and covering the entire field of view, in this case betweenlens 3105 and grating 3110. Characteristics of channel 3100, such as thefocal length of lens 3105, the spacing X between grating 3110 and array3115, the spatial frequency range of grating 3110, the depth of gratingfeatures and composition of grating 3110, and the geometry of thegrating sub-elements may be optimized for the selected color. An imageprocessor (not shown) can combine information collected by the colorchannels to recover a color image.

The color channels can have fixed lenses with distinct focal lengths, orcan have the same lens but distinct spacing between the lens andgrating. In cameras with multiple channels of diverse focal lengths, thechannel or channels presenting the sharpest focus may be used to capturehigh-resolution scene information, while the other, relatively defocusedchannels, can provide color information. Techniques to “paint on”relatively low resolution color information onto a high-resolution imageare well known to those of skill in the art.

Cameras with multiple channels, whether for the same or differentwavelength bands of interest, provide measures of parallax that can becombined with other information derived from orientation chirps to makea depth map of a scene. Also advantageous, information from multiplechannels can be used to disambiguate depth in the case where the objectexhibits deceptive structure. For example, a scene with in-focus spiralpatterns may appear defocussed to a spiral ensemble. In such cases across check to one or more additional channels can resolve ambiguity byselecting which “reality” is most consistent with the image dataprovided by the disparate channels

Methods and systems for detecting disparities between images of the samescene have wide applications in e.g. video surveillance and medicaldiagnosis. An image can be compared with a reference, such as to find adefect, or can be compared with a prior image of the same scene toidentify changes. The following discussion details image-changedetectors that compare successive images of the same scene to detectmotion. However, these detectors and the methods they employ can be putto other uses.

Some imaging applications, such as video surveillance, wasteconsiderable power and memory resources monitoring unchanging scenes. Toaddress this problem, some cameras support a low-power mode in which animage sensor's spatial and temporal resolutions are dramaticallyreduced. Fewer pixels are sensed, and less frequently, which saves powerat the sensor, and the relative paucity of data saves image-processingand transmission power. Image data collected in the low-power mode isused to detect changes, and detected changes can trigger the camera toswitch to a higher-performance mode that supports higher spatial andtemporal resolutions to verify the change detected in the low-power modeand/or capture an image.

FIG. 32 depicts an image-change detector 3200 that supports a low-powermode that is so efficient that it can be powered using (optionally) anintegrated photocell. Detector 3200 can be instantiated on a singlesemiconductor die, and in this embodiment includes a photodetector array3205 with overlying grating 1700, an analog-to-digital converter (ADC)3210 to convert analog information from array 3205 to digital data,logic and memory 3215 to process, store, and communicate the digitalsignals, and a photovoltaic cell 3220 to power each of the otherelements. Whether implemented on a single die or multiple die, thedetector 3200 can be packaged opaquely, with a transmissive windowoverlying the location of the grating 1700 (and the optionalphotovoltaic cell).

Detector 3200 supports a low-resolution, low-power sentinel mode tosense changes in scene activity, and a higher-resolution mode thatcaptures higher-resolution image data responsive to detected changes.Some embodiments support an additional mode or modes, as detailed below.Although logic and memory 3215 can support the change detection andimaging function, some implementations may have a main function that isdifferent from imaging, with the change detection providing an input tothe chip to add situational awareness. In such cases, it may not benecessary that photodetector array 3205 contain enough elements toproduce a higher-resolution image.

FIG. 33 depicts array 3205 of FIG. 33 as an array of pixels 3300. Eightnonadjacent pixels 3300 are darkened to indicate a subset that is activein the low-power mode. An orientation chirp 3305 represents a sharplyfocused response from an exemplary imaged point source as it may appearat the sensor plane. Chirp 3305 is illustrated as dark on a lightbackground for ease of illustration, but would appear as a relativelybright pattern. Chirp 3305 illuminates a set of R pixels within a convexhull 3310, the smallest convex set of S pixels that includes all theilluminated pixels. (Convex hull 3310 may be visualized as the shapeformed by a rubber band stretched around chirp 3305.) Chirp 3305provides a rich set of spatial modulations spanning the areas of hull3310 that dramatically increases motion sensitivity.

Conventional image sensors resolve a point source as a focused “dot” ona sensor array. If a camera is to detect very small movements, a richset of active pixels much be maintained even in a low-power modeImagine, for example, that a point source is resolved as a sharp orblurred dot on array 3205 such that only one or a collection ofneighboring pixels is illuminated. In that case, the point source couldmove considerably relative to the sensor without detection by any of theeight active pixels in this low power mode.

Chirp 3305, the result of illumination by a point source, includes“arms” of changing light intensity that illuminate many more pixels,including nonadjacent ones, than would a resolved spot, and that sweepacross a great many pixels 3300 as the point source moves relative tothe sensor. Consequently, fewer pixels 3300 need be polled to cover thevisual field than with a system employing traditional focusing optics.In this example, movement of the point source that moves chirp 3305 apixel or two in any direction within the X-Y plane would impact at leastone of the eight active pixels 3300, and could thus be sensed. Sensingmay involve analog-to-digital conversions of the signals from the samesubset of photodiodes at different points in time. In other embodiments,analog sample-and-hold circuits and comparators can be used to signalchanges in the imaged field of view. Depending upon the application,such sensed motion could be the information of interest, or could beused to bring detector 3200 out of the low-power mode to take and storeone or more sets of relatively high resolution data.

Some embodiments support additional operational modes, or “stages.” Inone embodiment, for example, logic and memory 3215 support a three-statemachine comprising a sentinel stage, a tentative stage, and aconfirmation stage that use progressively larger sets of image data tosense and confirm changes in an imaged scene. In the sentinel stage, n1pixels are monitored to generate compact image frames, relatively smallsets of image data that represent considerably lower image resolutionthan the maximum available from the sensor. If k1 (≦n1) of the pixelvalues change by a criterion value θ₁ between successive image frames,then the state machine transitions to the tentative stage.

In the tentative stage, n2 pixels are monitored to generate frames ofhigher resolution than those generated in the sentinel stage but stillless than the full field available from the sensor. If k2 (≦n2) of thepixel values change by a criterion value θ₂ between successive frames,then the state machine transitions to the confirmation stage; otherwisethe system reverts to the sentinel stage. If the system is in theconfirmation stage, n3 pixels are monitored and if k3 (≦n3) of thesepixels change by a criterion value θ₃ between successive frames, thenthe state machine emits a signal denoting image change detected andremains in the confirmation stage. In this example, the frames capturedin the confirmation mode represent the full resolution of the imagesensor. Detector 3200 can take some action responsive to detectedmotion, such as to capture and invert an image, or to alert someexternal system to the detected motion. From the confirmation stage,detector 3200 can transition back to the tentative or sentinel stage tosave power, or responsive to the absence of detected motion.

One benefit of this system is that, because of the grating optics, eachphotodetector pixel responds to a range of positions in the field ofview; thus the number of pixels that needs be monitored is lower(dissipating lower power) than in a traditional lens-based system, inwhich each pixel responds to a very small range of positions in thefield of view. Circuit analyses show that some ADC embodiments canobtain sub-400 nW image change detection, with the power required of ADC3210 dominating. Address generator circuits for polling subsets ofpixels in support of reduced power consumption are well known to thoseof skill in the art, so a detailed discussion is omitted.

FIG. 34 depicts array 3205 of FIG. 33 with twenty-five nonadjacentpixels 3300 darkened to indicate a subset n2 that is active in anexample of a tentative mode in which an increased number of pixels arepolled for increased motion sensitivity. Though not apparent in thisimage, sensitivity can also be changed by e.g. adjustingintensity-change thresholds and frame rates. Should motion be detectedin this mode, the imaging device can enter a confirmation stage in whichall pixels 3300 are polled. The resultant frames can then be used toconfirm motion, and can be used to produce images that can be understoodby a human observer. The resultant frames can be stored locally and readlater, or can be transmitted to a remote device. The production ofimages from human observers can likewise be done locally or remotely. Insome embodiments confirmed motion can activate and direct a separate,higher-resolution imaging system.

FIG. 35 depicts response 1930 of FIG. 19 as an exemplary chirp thatdefines a convex hull 3500. To find the convex hull for a given imagingdevice, response 1930 can be captured by the sensor array. With thebrightest pixel(s) serving as a reference, those pixels with at least10% of that maximum brightness are included in that set of pixel valuesrepresentative of the response. Convex hull 3500 is the smallest convexset of pixels that includes that set of pixel values. The convex hull isnot used for image acquisition or analysis, but affords a measure ofresponse area that can be used to characterize the ratio of activepixels relative to the response and the richness of spatial modulations.In this example, response 1930 includes many and diverse spiral “arms”that collectively provide much more information for detecting motionthan would a focused spot.

The examples of FIGS. 33-35 include but one point-spread response, andthe response is nearly as large as the pixel array. Detectors inaccordance with other embodiments can have more of fewer responses, andeach response can occupy a larger or smaller convex hull. For example,an image detector can employ a tessellated optical element like that ofFIG. 24 to produce a point-spread response that exhibits a web offeatures that can be distributed over all or a portion of an underlyingarray.

FIG. 36 is a flowchart 3600 illustrating a method of detecting apparentmotion in accordance with one embodiment of detector 3200 of FIG. 32.Photovoltaic cell 3220 provides sufficient power to support a sentinelmode in ambient light, with enough extra to charge integrated orexternal energy-storage devices capable of supporting bursts of use inhigher-performance modes. In some embodiments detector 3200 includes aseparate or integrated RFID chip and associated antenna to allow imagedata to be retrieved wirelessly. Detector 3200 can support other formsof wired or wireless connections, as will be understood by those ofskill in the art. An example of array 3205 with a grating 1700 isdetailed in an accompanying document entitled “Ultra-Miniature Low-PowerLensless Image Change Detector,” which is incorporated herein byreference.

However detector 3200 is powered, logic and memory 3215 (hereafter“logic” 3215) automatically enters the low-power sentinel mode on powerup (3605). Once in this mode, logic 3215 repeatedly polls n1 pixels, asmall subset of the total available in array 3205, to produce smallframes of image data (3610/3615). Successive frames are then compared toidentify differences.

For any successive pair of frames, per a decision 3620, if some or acombination of corresponding pixel values exhibit intensity changes Δθthat exceed a threshold T1, then logic 3215 enters a tentative mode inwhich logic 3215 repeatedly polls n2 pixels, wherein n2>n1 but is stilla subset of the total available in array 3205, to produce larger framesof image data (3625/3630).

Per decision 3633, device 3200 determines whether some or a combinationof corresponding pixel values from the successive frames taken in thetentative mode exhibit intensity changes AO that exceed a threshold T2.Device 3200 can remain in the tentative mode for some number of frameswithout intensity changes meeting threshold T2, but will eventuallyreturn to the sentinel state to save power. Should threshold T2 be met,logic 3215 enters a more power-intensive confirmation mode in whichlogic 3215 repeatedly polls n3 pixels (3635/3640).

In this example, the value n3 is the total available in array 3205 andthus produces full frames of image data. For any successive pair of fullframes, per a decision 3645, if some or a combination of correspondingpixel values exhibit intensity changes Δθ that exceed a third thresholdT3, then logic 3215 issues a signal confirming detection of movement(3650). Device 3200 can remain in the confirmation mode for some numberof frames without intensity changes meeting threshold T3, but willeventually return to the sentinel mode to save power. In otherembodiments device 3200 can transition to the tentative mode beforecontinuing to the sentinel mode. In embodiments that are power limited,device 3200 can enter a standby mode or one of the lower-power modesdespite detected motion to allow time to generate or store sufficientpower to return to the task of motion detection.

The thresholds T1, T2, and T3 used in the different modes can bedifferent, each tailored for the needs of the different modes. Also, thedifferent modes can be accompanied by changes in e.g. frame rate and theintegration time employed by ADC 3210 and/or logic 3215 to acquire imagedata.

Comparisons of successive sets of image data are not limited to just twosets, or to adjacent sets. For example, frames can be averaged orotherwise combined to reduce the effects of noise, and individual oraveraged sets of frames can be compared with earlier individual oraveraged sets of frames. Change detection may also be based oncomparisons with multiple distinct reference image data. For example, animage sensor experiencing motion due to e.g. wind may produce a largebut finite number of distinct reference frames or sub-frames thatrepresent no movement within a scene. The same might be true where winddisturbs elements of scene (e.g., a waving branch). A detector mightlearn such sets and indicate change only when the most recent set ofimage data fails to match any in the preceding sets.

Array 3205 and grating 1700 can be created using standard CMOSprocesses, and its formation is thus compatible with any number offunctional blocks. Virtually any integrated circuit that might benefitby inclusion of an imaging device can be adapted to include one. Forexample, a technology referred to as “smartdust” describes systems inwhich many small electromechanical systems (MEMS) can be operated on adistributed, wireless computer network to collectively perform varioussensing and communication tasks. Smartdust devices can be on the orderof a few millimeters on a side, which is easily sufficient toincorporate a sensor of the type detailed herein. In one embodiment, forexample, the inventors created a 128×128-pixel sensor that is 200microns on a side. Image sensors of the type detailed herein can beintegrated so inexpensively that they can be incorporated into creditcards and other forms of identification for security purposes, or tofacilitate vision systems in the field of microrobotics.

The foregoing detectors support modes that poll different numbers ofpixels to provide different tradeoffs between power usage andresolution. Other embodiments can support more, fewer, and differentmodes. For example, and an image-change detector can have but one modeusing all available pixels, but could nevertheless support modes thatemploy different thresholds, framerates, integration periods, sweeprate, etc.

Some lighting fixtures produce not a steady illumination, but a flicker,often at the frequency of the AC current powering them, or at twice thisfrequency. Many applications would benefit from being able to rejectthis flicker, yet stay sensitive to the motion or appearance of othertypes of light sources. The selective blindness to repeating flicker dueto illumination periodicity can be achieved by any of the following fourmeans. First, a bulk photoelement (a photodiode or photovoltaic) withoutany angle-selective optics over it is sensitive to the overallbrightness of the scene. If the integration period of each of the activepixels is not governed by time per se, but rather governed by crossing athreshold in the accumulated photocurrent incident on the bulkphotoelement, then the signals observed at each active pixel is scaledby the overall brightness of the room, which will be sensitive toflicker in a way similar to the active pixels. Second, the integrationperiod of the active pixels can be set to an integer multiple of therepetition period of the flickering illumination. Third, the integrationperiods for any two signals that are to be compared can commence at thesame phase of the flickering illumination. Fourth, the signals from eachactive pixel can be pooled to arrive at an estimate of the overallbrightness, and individual signals can first be normalized by thisbrightness before subsequent normalized frames are compared.

While the subject matter has been described in connection with specificembodiments, other embodiments are also envisioned. For example; whileeach grating detailed previously may be used in connection withphotoreceptors to collect incident light, gratings in accordance withthese and other embodiments can be used more generally in imagingdevices that project images from photo-emitters rather than or inaddition to sensing them; cameras described as using lenses could alsoemploy other types of optical elements (e.g., minors); the wavelengthband of interest can be broader or narrower than the visible spectrum,may be wholly or partially outside the visible spectrum, and may bediscontinuous; and cameras and gratings detailed herein can be adaptedfor use in multi-aperture or programmable-aperture applications. Thewavelength band of interest is the visible spectrum in these examples.Other variations will be evident to those of skill in the art.Therefore, the spirit and scope of the appended claims should not belimited to the foregoing description. Only those claims specificallyreciting “means for” or “step for” should be construed in the mannerrequired under the sixth paragraph of 35 U.S.C. §112.

What is claimed is:
 1. An image-change detector comprising: aphotodetector array of pixels; a phase grating to receive incidentlight, the phase grating to induce near-field spatial modulationsresponsive to a point source, the modulations illuminating a patternover nonadjacent ones of the pixels at the photodetector arrayresponsive to the incident light; and logic coupled to the photodetectorarray to successively poll the pixels to produce sets of image data, andto compare the sets.
 2. The detector of claim 1, the logic to poll afirst number of the pixels in a low-power mode and a second number ofthe pixels higher than the first number is a second mode.
 3. Thedetector of claim 1, wherein the spatial modulations comprise anorientation chirp.
 4. The detector of claim 3, wherein the orientationchirp is invertible.
 5. The detector of claim 1, the phase gratinghaving a minimum separation from the photodetector array of no more than400 times a maximum wavelength of interest.
 6. The detector of claim 1,wherein the near-field spatial modulations are substantially wavelengthindependent over a wavelength band of interest.
 7. The detector of claim1, wherein the near-field spatial modulations exhibit radial lines. 8.The detector of claim 1, further comprising a photovoltaic cellintegrated with the photodetector and the logic to power thephotodetector and the logic.
 9. An image-change detector comprising: aphotodetector array of pixels; an optical element to receive incidentlight, the optical element exhibiting a point-spread responseilluminating a pattern over R of the pixels and defining a convex hullenveloping an area of S>2R pixels of the photodetector array; and logiccoupled to the photodetector array to successively poll the pixels toproduce sets of image data, and to compare the sets.
 10. Theimage-change detector of claim 9, the logic to poll n1<S/2 of the pixelsin a first mode and to poll greater than n1 of the pixels in a secondmode.
 9. The image-change detector of claim 8x1, wherein the polling inthe first mode produces successive sets of pixel values, the logic totransition from the first mode to the second mode responsive tointensity changes to the successive sets.
 11. The image-change detectorof claim 10, wherein the logic polls the S pixels in the second mode.12. The image-change detector of claim 10, the logic to poll n2 of thepixels in the second mode, where n2<S, wherein the polling the n2 pixelsproduces successive sets of n2 pixel values, the logic to transitionfrom the second mode to a third mode responsive to intensity changes inthe successive sets.
 13. The image-change detector of claim 12, thelogic to transition from the second ode to the first mode if theintensity changes are below a threshold.
 14. The image-change detectorof claim 10, wherein the logic polls the n1 of the pixels at a firstframe rate in the first mode and polls the greater than n1 of the pixelsat a second frame rate in the second mode.
 15. The image-change detectorof claim 10, wherein the logic polls the n1 of the pixels over a firstpixel integration time in the first mode and polls the greater than n1of the pixels at a second pixel integration time in the second mode. 16.The image-change detector of claim 9, wherein the point-spread responsecomprises a spiral intensity pattern.
 17. The image-change detector ofclaim 9, the logic to poll n1<S/2 of the pixels in the convex hull in afirst mode and to poll greater than n1 of the pixels in the convex hullin a second mode.
 18. A method of detecting a change in a scene, themethod comprising: receiving light in a wavelength band of interest fromthe scene at an optical element, the optical element exhibiting apoint-spread response on an array of pixels, the point-spread responseilluminating a pattern over R of the pixels and defining a convex hullenveloping an area of S>2R pixels; repeatedly polling n1<R/2 of thepixels in a first mode, to produce first sets of image data; comparingthe first sets of image data to identify a difference; if the differenceexceeds a threshold, transitioning to a second mode and polling n2>n1pixels; and if the difference is below the threshold, remaining in thefirst mode.
 19. The method of claim 18, further comprising: repeatingthe polling of the n2 pixels in the second mode to produce second setsof image data; comparing the second sets of image data to identify asecond difference; and if the second difference exceeds a secondthreshold, transitioning to a third mode and polling n3>n2 pixels. 20.The method of claim 18, further comprising transitioning to the firstmode if the second difference is below the threshold.
 21. The method ofclaim 18, further comprising capturing the first sets at a first setrate in the first mode and capturing the second sets at a second framerate different than the first frame rate in the second mode.
 22. Themethod of claim 18, further comprising capturing each of the first setsover a first integration time in the first mode and capturing each ofthe second sets over a second integration time different than the firstintegration time in the second mode.
 23. The method of claim 18, whereinthe n1 pixels and the n2 pixels are within the convex hull.